usage: convert_tokenizer [-h] [-o OUTPUT] [--with-detokenizer] [--skip_special_tokens] [--use-fast-false] [--trust-remote-code]
                         [--tokenizer-output-type {i32,i64}] [--detokenizer-input-type {i32,i64}] [--streaming-detokenizer STREAMING_DETOKENIZER]
                         name

Converts tokenizers from Huggingface Hub to OpenVINO Tokenizer model.

positional arguments:
  name                  The model id of a tokenizer hosted inside a model repo on huggingface.co or a path to a saved Huggingface tokenizer
                        directory

options:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output directory
  --with-detokenizer    Add a detokenizer model to the output
  --skip_special_tokens
                        Produce detokenizer that will skip special tokens during decoding, similar to huggingface_tokenizer.decode(token_ids,
                        skip_special_tokens=True).
  --use-fast-false      Pass `use_fast=False` to `AutoTokenizer.from_pretrained`. It will initialize legacy HuggingFace tokenizer and then
                        converts it to OpenVINO. Might result in slightly different tokenizer. See models with _slow suffix https://github.com/op
                        envinotoolkit/openvino_contrib/tree/master/modules/custom_operations/user_ie_extensions/tokenizer/python#coverage-by-
                        model-type to check the potential difference between original and OpenVINO tokenizers
  --trust-remote-code   Pass `trust_remote_code=True` to `AutoTokenizer.from_pretrained`. It will execute code present on the Hub on your local
                        machine
  --tokenizer-output-type {i32,i64}
                        Type of the output tensors for tokenizer
  --detokenizer-input-type {i32,i64}
                        Type of the input tensor for detokenizer
  --streaming-detokenizer STREAMING_DETOKENIZER
                        [Experimental] Modify SentencePiece based detokenizer to keep spaces leading space. Can be used to stream a model output
                        without TextStreamer buffer